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SELECTION  AND  RANKING  PROCEDURES:  A BRIEF  INTRODUCTION* 

Shanti  S.  Gupta 
Purdue  University 


1.  ORIGIN  OF  THE  PROBLEM 
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A common  problem  faced  by  an  experimenter  is  one  of  comparing 

several  categories  or  populations.  These  may  be,  for  example, 

different  varieties  of  a grain,  different  competing  manufacturing 

processes  for  an  industrial  product,  or  different  drugs  (treatments) 

for  a specific  disease.  In  other  words,  we  have  k(^2)  populations 

and  each  population  is  characterized  by  the  value  of  a parameter  of 

interest  0,  which  may  be,  in  the  example  of  drugs,  an  appropriate 

measure  of  the  effectiveness  of  a drug.  The  classical  approach  to 

this  problem  is  to  test  the  homogeneity  (null)  hypothesis  Hq*. 

®1  " ~ ®re  the  values  of  the  parameter 

for  these  populations.  In  the  case  of  normal  populations  with 

2 

means  0j,...,0j^  and  a common  variance  a , the  test  can  be  carried 
out  using  the  F-ratio  of  the  analysis  of  variance. 

The  above  classical  approach  is  inadequate  and  unrealistic  in 
the  sense  that  it  is  not  formulated  in  a way  to  answer  the 
experimenter's  question,  namely,  how  to  identify  the  best  category? 

In  fact,  the  method  of  least  significant  differences  based  on  ! 

t-tests  has  been  used  in  the  past  to  detect  differences  between  the  | 

average  yields  of  different  varieties  and  thereby  choose  the  'best'  1 

j 

variety.  But  this  method  is  indirect,  less  efficient  and  does  not  I 
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easily  provide  an  overall  probability  of  a correct  selection.  Also 
the  multiple  comparison  techniques  developed  largely  by  Tukey  (1949) 
and  Scheffe  (1953)  arose  from  the  desire  to  draw  inference  about  the 
populations  when  the  homogeneity  hypothesis  is  rejected.  For 
details  of  several  multiple  comparison  techniques,  see  Miller  (1966). 

2.  SELECTION  AND  RANKING  PROCEDURES 

The  formulation  of  a k-sample  problem  as  a multiple  decision 
problem  enables  the  experimenter  to  anwer  his  natural  questions 
regarding  the  best  category.  Among  the  early  investigators  of  such 
procedures  are  Paulson  (1949),  Bahadur  (1950),  Bahadur  and  Robbins 
(1950).  The  formulation  of  multiple  decision  procedures  in  the 
framework  of  selection  and  ranking  procedures  has  been  generally 
accomplished  either  using  the  indifference  zone  approach  or  the 
(random  sized)  subset  selection  approach.  The  former  approach  was 
introduced  by  Bechhofer  (1954).  Substantial  contribution  to  the 
early  and  subsequent  developments  in  the  subset  selection  theory 
has  been  made  by  Gupta  starting  from  his  work  in  1956. 

3.  DESCRIPTION  OF  THE  TWO  APPROACHES 

Bechhofer  (1954)  considered  the  problem  of  ranking  k normal 

means.  In  order  to  explain  the  basic  formulation,  consider  the 

problem  of  selecting  the  population  with  the  largest  mean  from  k 

normal  populations  with  unknown  means  y.,  i=l,...,k,  and  a common 
2 — ^ 

known  variance  o . Let  i=l,...,k,  denote  the  means  of 
independent  samples  of  size  n from  these  populations.  The  'natural' 
procedure  (which  can  be  shown  to  have  optimum  properties)  will  be  to 
select  the  population  that  yields  the  largest  x^.  The  experimenter 
would,  of  course,  need  a guarantee  that  this  procedure  will  pick  the 
population  with  the  largest  with  a probability  not  less  than  a 
specified  level  P*.  For  the  problem  to  be  meaningful  P*  lies 
between  1/k  and  1.  Since  we  do  not  know  the  true  configuration  of 
the  y^,  we  look  for  the  least  favorable  configuration  (LFC)  for 
which  the  probability  of  a correct  selection  (PCS)  will  be  at  least 


t 


P*.  Since  the  LFC  is  given  by  = ...  = the  probability 
guarantee  cannot  be  met  whatever  be  the  sample  size  n. 

A natural  modification  is  to  insist  on  the  minimum  probability 
guarantee  whenever  the  best  population  is  sufficiently  superior  to 
the  next  best.  In  other  words,  the  experimenter  specifies  a 
positive  constant  A*  and  requires  that  the  PCS  is  at  least  P* 
whenever  ^ where  p^^j  ^ ^ denote  the 

ordered  means.  Now  the  minimization  of  PCS  is  over  the  part  of 


the  parameter  space  in  which  p 


Ik] 


"'[k-1] 


The  complement  of 


is  called  the  indifference  zone  for  the  obvious  reason.  The  LFC 
in  is  given  by  = ...  = "^be  problem  is  to 

determine  the  minimum  sample  size  required  in  order  to  have  PCS  > P* 
for  the  LFC. 

Bechhofer's  formulation  is  more  general  than  what  is  describ  d 
above.  His  general  ranking  problem  includes,  for  example,  selection 
of  the  t best  populations. 

In  the  subset  selection  approach,  the  goal  is  to  select  a 
non-empty  subset  of  the  populations  so  as  to  include  the  best 
population.  Here  the  size  of  the  selected  subset  is  random  and  is 
determined  by  the  observations  themselves.  In  the  case  of  normal 

2 

populations  with  unknown  means  and  a common  variance  a , 

the  rule  proposed  by  Gupta  (1956)  selects  the  population  that  yields 
X.  if  and  only  if  x.  > max  3T.  - — , where  d = d(k,P*)  > 0 is 

determined  so  that  the  PCS  is  at  least  P*.  Here  a correct  selection 
is  selection  of  any  subset  that  includes  the  population  with  the 
largest  Thus,  the  LFC  is  with  regard  to  the  whole  parameter 

space  R.  Under  this  formulation,  for  given  k and  P*  we  determine  d. 
The  rule  explicitly  involves  n.  In  general,  the  rule  will  involve 
a constant  which  depends  on  k,  P*,  and  n.  The  performance  of  a 
subset  selection  procedure  is  studied  by  evaluating  the  expected 
subset  size  and  its  supremum  over  the  parameter  space  G. 


4.  TWO  EXAMPLES 


Example  1.  Given  five  normal  populations  with  unknown  means 
and  a common  variance  64,  it  is  desired  to  find  which  population 
has  the  largest  mean  and  to  guarantee  that  the  probability  of 
correctly  choosing  that  population  is  at  least  0.90  whenever 
P[g]  - 4.  How  many  observations  must  be  taken  from  each 

population? 

We  have  \ = 2.5997  from  Table  I of  Bechhofer  (1954)  where 
X = A*/o  = 0.5.  Thus  we  take  28  observations  from  each  population. 

Example  2.  Given  the  five  normal  populations  as  above,  it  is 
desired  to  select  a non-empty  subset  of  these  populations  based  on 
n=8  observations  from  each  population  with  the  guarantee  that  the 
population  with  the  largest  mean  will  be  included  in  the  selected 
subset  with  a probability  not  less  than  0.90.  Using  Tables  in 
Gupta  (1963)  (or  Gupta,  Nagel  and  Panchapakesan  (1973),  we  find 
that  d=d(5,.90)  = 2.5997  and  d/^n  = .983.  Using  a program  for 
generating  random  normal  deviates  N(0  + o6,  1),  6=1,  a=0,l,2,3,4, 
the  following  sample  means  x^^  based  on  n=7  were  observed: 

-0.1940  .7987  2.5953  2.8754  4.3841 

In  this  example  then,  the  subset  selection  rule  selects  only  the 
observed  populations  corresponding  to  observed  x.  in  the  interval 
[3.401,  4.384].  Thus,  only  the  population  with  the  largest  x^ 
value  is  selected  in  the  subset.  Note  the  procedure  of  Bechhofer 
will  also  select  the  same  population  with  probability  of  a correct 
selection  equal  to  .969.  The  subset  selection  procedure  gives  the 
probability  of  a correct  selection  to  be  between  .999  and  1.000  and 
the  associated  expected  proportions  of  the  number  of  populations 
selected  in  this  case  lies  between  .28  and  .36. 

5.  MODIFICATIONS  AND  GENERALIZATIONS 

The  above  basic  formulations  have  been  modified  and  generalized 
by  several  authors.  Mahamunulu  (1966)  has  discussed  a generalized 
goal  for  fixed-size  subset  selection.  His  goal  is  to  select  a 
subset  of  size  s from  k populations  so  that  the  selected  subset 
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contains  at  least  c of  the  t best  populations.  Of  course,  the 
constants  c,s,t,k  should  satisfy  some  obvious  inequalities.  The 
problem  is  to  determine  the  minimum  sample  size  such  that  the  PCS 
^ P*  whenever  ^ d*  (in  the  case  of  normal  means). 

Desu  and  Sobel  (1968)  considered  the  inverse  problem  of  selecting  a 
subset  of  the  smallest  fixed  size  s given  the  sample  size  so  that 
the  selected  subset  will  contain  the  t best  of  k populations 
(t  £ s ^ k).  A formulation  for  eliminating  strictly  inferior 
populations  has  been  used  by  Desu  (1970),  and  Carroll,  Gupta  and 
Huang  (1975).  Some  generalized  results  in  this  direction  are  given 
by  Panchapakesan  and  Santner  (1977). 

A restricted  subset  size  formulation  has  been  studied  by 
Santner  (1975),  and  Gupta  and  Santner  (1973).  The  idea  here  is  to 
select  a subset  of  random  size  subject  to  this  size  not  exceeding  a 

• maximum.  In  the  case  of  k normal  populations  of  unknown  means  g., 

• 2 ^ 
1=1,. ..,k,  and  a common  known  variance  a , let  m(l  ^ m ^ k)  be  the 

maximum  subset  size  permissible.  The  goal  is  to  select  a subset  of 

size  not  exceeding  m such  that  the  subset  contains  the  population 

with  the  largest  mean  with  a probability  not  less  than  P*  whenever 

^ rule  proposed  by  Gupta  and  Santner  (1973) 

selects  the  population  corresponding  to  x^  if  and  only  if 

7i  > max  xj^j  - 

where  d > 0 is  a constant  to  be  suitably  determined.  For  given  6, 
n,  and  m,  the  probability  guarantee  can  be  met  for  P*  values  not 
exceeding  a certain  value  P^  = P^(k,m,n,6). 

Other  generalizations  and  modifications  have  been  studied  by 
Deverman  and  Gupta  (1969),  Sobel  (1969),  Gupta  and  Panchapakesan 
(1972) , Alam  and  Thompson  (1973) , and  Huang  and  Panchapakesan 
(1976).  There  has  also  been  interest  in  decision-theoretic 
formulations  of  subset  selection.  Specific  mention  should  be  made 
of  Studden  (1967),  Deely  and  Gupta  (1968),  Goel  and  Rubin  (1975), 
Bickel  and  Yahav  (1977),  Berger  (1977),  and  Hsu  (1977).  In  the 
papers  by  Chemoff  and  Yahav  (1977)  and  Hsu  (1977),  Monte  Carlo 
studies  have  been  carried  out  in  the  framework  of  subset  selection 
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which  show  that  Bayes  procedures  can  be  closely  approximated  by 
Gupta  type  procedures. 

6.  CONCLUSION 

In  the  last  twenty-five  years,  the  research  in  the  area  of 
selection  and  ranking  procedures  has  progressed  steadily  and  the 
present  count  of  published  papers  and  technical  reports  exceeds 
five  hundred.  Though  these  procedures  have  the  potential  for 
application  and  the  use  is  increasing,  it  should  be  admitted  that 
such  use  is  not  yet  on  a large  scale.  We  should  perhaps  hasten  to 
add  that  the  situation  is  not  unusual  considering  the  fact  that  it 
calls  for  giving  up  the  ingrained  habit  of  testing  of  hypotheses  or 
tests  of  significance  and  ANOVA  on  the  part  of  applied  statisticians. 
The  time  is  right  to  remind  the  theoreticians  among  ourselves  that 
the  gap  in  the  communication  with  the  users  is  yet  to  be  closed. 

Some  encouraging  signs  of  adopting  multiple  decision  (selection 
and  ranking)  theory  as  realistic  alternative  to  hypotheses  testing 
have  again  appeared  on  the  horizon.  At  an  international 
symposium  at  Purdue  in  May  1976,  there  were  papers  presented  on 
this  topic  from  some  competent  statisticians,  who  had  earlier  not 
worked  on  these  problems.  These  include  Bickel,  Chernoff  and  Yahav. 
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